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suffering from roles/relationships confusion among the major state work force 
partners; disproportionately focused on process not outcomes or impacts; 
emphasizing participant results to near total exclusion of employer-oriented 
measures and management or evaluation mechanisms; ad hoc evaluations whose 
findings have been poorly integrated into policy decision making; and very 
uneven across various programs and agencies. Steps recommended to move 
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Executive Summary 

Texas is in the process of shifting from a categorical, program-specific approach 
to providing workforce services for its residents and employers to one which is integrated 
and systems oriented. This shift follows from the passage and subsequent implementation 
of two major pieces of state legislation: Senate Bill 642, the Texas Workforce and 
Economic Competitiveness Act of 1993; and House Bill 1863, which amended it in 1995. 
Such a shift requires a fundamentally different approach to evaluating its workforce 
development effort. 

Accountability in an Integrated Workforce System 

Within an integrated workforce system, all available resources are guided by a 
single mission and vision, and implementation strategies are clearly focused on achieving 
common results. This integrated service delivery system must be supported by a 
management system which includes all the traditional management functions — planning, 
organizing, staffing, operating and controlling — and all of the necessary connections 
among them, i.e., critical feedback loops and the data collection and reporting systems 
required for managing such a system. 

In the broadest sense, accountability means being able to ensure taxpayers, as well 
as both major workforce development system customers — residents (e.g., students, actual 
and potential workers) and employers — and society as a whole, that public investments 
are being prudently managed and are yielding positive returns to both society and those 
being served. 

The Current System 

The current accountability system for Texas workforce development services can 
be characterized as: largely programmatic and categorical; driven for the most part by 
federal legislative and regulatory requirements; focused primarily on performance 
management rather than evaluation; founded upon an artificial and inappropriate 
distinction between systems and program goals; suffering from roles/relationships 
confusion among the major state workforce partners, e.g., the Texas Council on 
Workforce and Economic Competitiveness (TCWEC) v. the Texas Workforce 
Commission (TWC), the Governor's Office (GO) and Legislative Budget Board (LBB) v. 
TWC and other administering agencies; disproportionately focused on process not 
outcomes or impacts; overwhelmingly emphasizing participant results to the near total 
exclusion of employer-oriented measures and management/evaluation mechanisms; 
evaluations, where they have been done, have been ad hoc and their findings poorly 
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integrated into mainline state policy decisionmaking at the highest levels; and very 
uneven across the various programs and agencies. 

Major Gaps and Challenges 

There are a number of major gaps in the current accountability system, some of 
the more important of which include: workforce service and agency boundary issues; 
strategic plan discrepancies; TCWEC authority issues; the independence and funding 
mechanisms for the Texas State Occupational Information Committee (SOICC), which 
has been serving as the key entity linking the various administrative data sets for 
management and evaluation; and the need for revisions to the existing core performance 
measures. 

Recommended Next Steps 

A number of important next steps are recommended to move towards a systems 
evaluation approach to accountability for Texas workforce development services. These 
recommended steps are organized into three broad phases. Phase I consists of steps to be 
taken immediately, i.e., before June 30, 1997. Phase II offers a series of longer-term steps 
to be taken from July 1997 through June 1999. And, Phase III presents steps which 
should be taken over the long run, between July 1999 and June 2001. This approach 
recognizes the immediate pressures to set in place some measures that will allow the 
system to respond to the elected and appointed officials who inevitably ask the 
straightforward questions, "Is it working?" and "Are we making a difference?" 
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Evaluation Action Plan 

Phase I: Immediate (present - June 1997) 

1.1 Establish the Entered Employment Rate and Average Earnings Gain 
Based on Previous Earnings as the key workforce measures for the 
next Biennium, for all workforce operating agencies. (LBB, GO & 
TCWEC) 

1.2 Design and establish performance "targets" for Entered Employment 
Rate and Average Earnings Gain measures. (TCWEC, partner 
agencies) 

1.3 Modify the existing core performance measures, eliminating several 
and making technical corrections to others. (TCWEC, partner 
agencies) 

1.4 Design the remaining core measures, including access/equity, 
employer-oriented, skill attainment and cost-effectiveness (e.g., retum- 
on-investment) measures. (TCWEC, partner agencies) 

1.5 Identify the basic data reporting elements required to support the core 
workforce performance measures. (TCWEC, partner agencies) 

1.6 Reinstitute SOlCC's status as an independent entity and provide for its 
funding. (Legislature) 

Phase II: Longer-Run (July 1997 - June 1999) 

2.1 Clarify key actors' strategic roles and relationships in the workforce 
system. 

(TCWEC) 

2.2 Redo the workforce system's strategic planning process, substantially 
revising existing goals/objectives. (TCWEC, state/local partners) 

2.3 Bring the state and local workforce planning, budgeting and funding 
process in line with the state's biennial planning and budgeting cycle. 
(LBB, GO, TCWEC, state/local partners) 




3/97 

3/97 

6/97 

6/97 

6/97 

6/97 

10/97 

12/97 

6/99 
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2.4 Refine the remaining core measures, including employer-based, 
access/ equity, skill attainment and program advancement. (TCWEC, 
state/local partners) 

2.5 Identify the basic data reporting elements required to support 
additional core performance measures. (TCWEC, state/local partners) 

2.6 Develop better mechanisms for communicating service goals, 
objectives and measures to local boards et al. (TCWEC, partner 
agencies) 

Phase HI: Long-Run (July 1999 - June 2001) 

3.1 Appropriate all federal and state workforce funds through TCWEC to 
workforce agencies and provide TCWEC with limited authority to 
modify agencies' allocations to achieve system objectives. 
(Legislature) 

3.2 Reserve 5-10 percent of appropriated federal and state workforce 
funds for rewarding state agencies and their local counterparts based 
on their contributions to system performance. (Legislature) 




1/98 

6/98 

6/99 

MM/99 

MM/99 



x 
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L INTRODUCTION 

Texas is in the process of shifting from a categorical, program-specific approach 
to providing workforce services for its residents and employers to one which is integrated 
and systems oriented. This shift follows from the passage and subsequent 
implementation of two major pieces of state legislation: Senate Bill 642, the Texas 
Workforce and Economic Competitiveness Act of 1993 and House Bill 1863, which was 
signed into law in 1995 amending the former act. Such a shift requires a fundamentally 
different approach- to evaluating its workforce development efforts in many important 
respects, many of which are characterized below. 

The purposes of this paper are to: 

• Present important concepts which underlie an ideal workforce delivery system 
and its accompanying accountability system; 

• Offer the outlines of an ideal workforce accountability system for Texas; 

• Summarize the status of the current accountability system for workforce services; 

• Outline major gaps and challenges in developing and implementing the new 
accountability system; and 

• Provide an systems evaluation action plan for moving Texas from the current to 
the ideal accountability system over the next few years. 
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II. SETTING THE FRAMEWORK 
Defining Workforce Development 

State law defines workforce development as encompassing two major service 
components: workforce education; and workforce training and services. The former 
generally refers to career-path programs that lead to some sort of recognized diploma, 
credential or associates degree. Workforce training and services are defined as other 
training and services related to maintaining and enhancing worker skills and productivity. 

In this context, workforce development can be broadly defined as all of those 
services related to developing, maintaining and utilizing the active or potential labor 
force (e.g., Bakke 1963; Harbison 1973). This definition includes preparation for work 
(K through 12 education) and further training for those employed as well as those 
unemployed. It is a life long process of both working and learning. Institutionally 
speaking, the workforce system encompasses a number of major subsystems that must be 
strategically woven together into the whole cloth of an integrated system. For public 
policy purposes, the primary subsystems are: the public education system; so-called 
"second-chance" services for those not realizing their productive potential in the labor 
market; and employer-sponsored training on the job or in the classroom. 

This definition is intentionally broad to give a sense of the breadth and depth of 
the workforce agenda. As a point of entry or a way to move forward with systems 
development efforts in Texas, the initial focus here is on labor exchange, postsecondary 
subbaccalaureate education, training and related services. This approach is also followed 
by a number nationally recognized education and training experts (e.g., Grubb 1996a, 
1996b). 

Accountability in an Integrated Workforce System 

Within an integrated workforce system, all available resources are guided by a 
single mission and vision, and implementation strategies are clearly focused on achieving 
common results (i.e., measurable objectives). This integrated service delivery system 
must be supported by a management system which includes all the traditional 
management functions — planning, organizing, staffing, operating and controlling — and 
all of the necessary connections among them, i.e., critical feedback loops and the data 
collection and reporting systems required for managing such a system (Mackenzie 1969). 
Accountability can be viewed as part of the traditional management control function at 
every level. 
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In the broadest sense, accountability means being able to ensure taxpayers, as well 
as both major workforce development system customers — residents (e.g., students, actual 
and potential workers) and employers — and society as a whole that public investments 
are being prudently managed and are yielding positive returns to both society and those 
being served. Thus, there are three broad components to an accountability system: fiscal 
accountability, compliance accountability, and what has been commonly referred to as 
program accountability, as follows: 

• fiscal accountability , i.e., making sure the dollars are spent for allowable 
purposes, in the appropriate time frame, etc., according to federal and state legal 
and regulatory requirements; 

• compliance accountability , where the concern is that services and the way they 
are provided comply with requirements of each of the various funding sources; 
and 

• program accountability — referred to here as services effectiveness and results 
accountability to reflect the shift from programs to services — is concerned with 
whether the services provided are making a difference, i.e., are performance 
expectations being met, both in terms of pre-established measures and standards 
and realizing a positive return on the public investment? 

It is this latter component which is the focus of this paper. Fiscal accountability is 
certainly necessary, a sentiment many former administrators and managers readily echo. 
But, fiscal accountability by itself, is not sufficient for ensuring an effective workforce 
development system. 

The basic framework within which a workforce accountability system should 
function is characterized in Figure 1. On the strategic level, broad goals and objectives 
are developed out of a highly structured, inclusive planning process, conducted at both 
the state and local levels. Strategic goals and objectives are established at a relatively 
high level of aggregation and represent broad statements of the direction workforce 
services should be leading for its customers. These are accompanied by a set of core 
systems performance measures and standards which are established in close relation to 
the strategic goals and objectives. At the strategic level, some type of rewards and 
sanctions process also must be in place to ensure that workforce performance has 
appropriate implications, whether positive or negative, at the strategic level. The 
principal mechanism for measuring and ensuring workforce service accountability 
strategically is systems evaluation. 
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Figure 1 

The Basic Framework 



Strategic 

Strategic Goals & Objectives 
Core Systems Measures & Standards 
Systems Evaluation 



Operational 

Operational Goals & Objectives 
Core Operational Measures & Standards 
Performance Management 



«— Program/Services Evaluation— » 



Rewards & Sanctions 



Rewards & Sanctions 



A completely parallel construct applies at the operational level, starting with a set 
of workforce goals and objectives. Operational goals and objectives translate broader 
strategic visions into more practical, day-to-day statements of direction which are 
measurable and potentially attainable for those responsible for workforce efforts at both 
the state and local level. Core performance measures and standards then relate to these 
goals and objectives and are implemented through the performance management process 
with appropriate rewards and sanctions. 

Strategic and operational accountability should be linked through the conduct of 
periodic service effectiveness/impact evaluations which seek to ensure that process and 
near-term performance standards applied at the operational level are valid indicators of 
longer-term net impacts for customers. For example, such evaluations attempt to 
determine whether and to what extent workforce job placement standards are correlated 
with long-term net impacts on earnings resulting from such services. These evaluations 
also offer important information for those responsible for administering and directing 
workforce service provision, including indications concerning the efficiency and 
effectiveness of particular service strategies (e.g., training, job search) in general and for 
population subgroups of special interest. To the extent possible, operational and systems 
accountability mechanisms should be complementary, moving state and local workforce 
efforts in the same direction. 

The framework offered above is developing. With few exceptions, workforce 
services historically have been provided in a program-specific, highly categorical 
environment. Most aspects of workforce-related services have been proscribed, from 
goals and objectives, to target populations and service strategies. Not surprisingly, the 
management and evaluation terminology which surrounds the field exhibits similar 
emphases and shortcomings. In recent years, a handful of efforts have been made to 
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address accountability in a more systemic manner (e.g., Bamow 1992; Blalock 1990, 
1996; King 1988; Rossi and Freeman 1993; and Trott and Baj 1996). But, as with all 
developmental efforts, the terminology is uneven and sometimes inconsistent. 
Experience with systems can be expected to improve the terminology and the sharpen the 
tools considerably over the next few years. 

Performance Management and Evaluation 

It is also helpful to distinguish between performance management on the one hand 
and evaluation on the other. These terms are widely used, too often interchangeably. 
Clearly defining -them should improve the’ discussion of accountability roles and 
relationships which follows. (Figure 2 provides some of the key distinguishing features 
of evaluation and performance management.) 

Rossi and Freeman (1993) state that comprehensive evaluation — the term most 
closely corresponding to the integrated workforce system context addressed here — can be 
defined as "[rjesearch and analysis covering the conceptualization and design of 
interventions, the monitoring of interventions, and the assessment of program utility" (p. 
2). Few evaluation treatments address the issue of evaluating an entire system, whether 
for delivering workforce or other services. Evaluation tends to be relatively 
comprehensive and attempts to address broad questions of design, service provision and 
impact where possible. It is often an activity which, by design, can only be performed 
over a period of several years, though some questions may be answered more quickly. 

Performance management, on the other hand, can be defined as a process utilized 
by administrators and managers for systematically measuring performance resulting from 
workforce interventions against a set of predetermined expectations and subsequently 
rewarding and/or sanctioning those intervening in order to achieve improved results. 
Performance management thus is an ongoing, day-to-day management function or 
process comprised of clearly articulated goals and objectives, a series of performance 
measures and their associated standards, a systematic methodology for establishing and 
adjusting the standards, and a series of rewards and/or sanctions (Bamow 1992). These 
elements should be firmly grounded in a clearly articulated performance management 
philosophy, which might ultimately aim to attain "world-class" performance, to foster 
continuous improvement, or to minimize substandard performance for example (Bamow 
and King 1996). Performance management provides continuous feedback to the 
managers, agencies and organizations responsible for monitoring the delivery of 
workforce services at the relevant level. Program monitoring is a component of 
performance management at the state and local level. 
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Figure 2: 

Performance Management v. Evaluation 



Performance Management 

• Used largely by program managers and 
administrators, at the local & state level 

• Focuses on services & their near- term (gross) 
outcomes 

• Features process & outcome elements 

• Short-run measurement period 

• High frequency of use 

• Relies on many, detailed measures 



Evaluation 

• Used largely by policymakers, administrators, 
planners & others, more at the state than local 
level 

• Focuses on broader issues of service strategies 
& their effects, including net impacts for key 
groups 

• Features process, outcome/impact & 
comprehensive systems elements 

• Longer-term measurement period 

• Periodic frequency of use 

• Uses few, broad measures 



Again, evaluation — whether systems or services, process or outcomes — and 
performance management function as integral, interrelated parts of an accountability 
system. They need to work in tandem and must be congruent, such that efforts found to 
be performing well against standards established for a performance management 
mechanism should also be associated with effective results for the wider system. Over 
time, the connection between these two accountability mechanisms needs to bdbolstered 
and reinforced. 

Types of Evaluation 

Several key types of evaluation are relevant to this discussion. Evaluations may 
be either systems or services evaluations, each of which may have both process and an 
outcomes (impact) dimensions. Moreover, benefit/cost or cost-effectiveness issues are 
often addressed in the course of evaluating workforce education, training and related 
efforts (Rossi and Freeman 1993; Gramlich 1990). To clarify further: 

Systems evaluation is defined here as systematically measuring and gauging the 
implementation, effectiveness and efficiency of the (workforce development) 
system as a whole, whether at the state or the local level. As such, it looks beyond 
specific services and their results, focusing instead on the results of interventions 
encompassing broad service strategies. 
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Blalock (1996) advises that systems evaluation is necessarily horizontal in nature 
and cannot address questions of net impact: a design which randomly assigns residents 
(or employers) to differing services may be feasible for an evaluation of a particular 
service, service strategy or program, but it is incapable of evaluating the effects of a 
system as a whole. 

Services evaluation is defined as systematically measuring and gauging the 
implementation, effects (including net impacts) and efficiency of particular 
services. 

Many designs can be developed for conducting such evaluations, depending on 
the particular questions at hand. Implementation designs may be needed to examine 
whether the services are being provided as planned and serving the designated groups. 
Gross outcome, quasi or true experimental designs may be called for to gauge issues of 
outcomes or net impacts of services. These may be coupled with benefit/cost analyses 
for addressing efficiency or resource use. 

Regardless of the type of evaluation or performance management, well designed, 
ongoing feedback loops to their respective audiences — policymakers for evaluation, and 
administrators and managers for performance management — are essential for 
continuously improving performance over time. 

Service Variation in a Universal Access System 

One of the more distinctive facets of the newly integrating workforce system is 
that both sets of customers, residents and employers, are offered universal access to the 
system. However, universal access does not mean that every customer is guaranteed 
access to every possible service offered. Every customer has access to some level of 
service, but the types of service provided are going to vary with the customer's needs 
and — at least until categorical federal and state programs give way to more flexible block 
grants — eligibility. Anyone should be able to access basic services offered by career 
centers offer (e.g., job vacancy information). Others will be guided to self-help services 
(e.g., job finding), while others with skills deficiencies might be referred to more 
intensive, developmental services. In essence, at least a three-pronged array of 
workforce service strategies should be available to customers depending upon their 
particular needs and other factors. As will be evident shortly, this inherent tension 
between universal access and needs-based service strategies makes the task of devising 
simple access/equity performance measures for more difficult than it has been under a 
traditional programmatic regime. 
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The Marbled Cake Analogy 

An analogy may prove helpful to better understand some of these concepts and 
their applicability. In essence, the emerging system offers governance and management 
responsibilities which resemble the texture of a marbled — as distinct from a layer — cake. 
In an ideal intergovernmental workforce system, governance and management 
responsibilities are not the exclusive right of any one level of government. Influence and 
control are shared, in this case both between state and local government, and among the 
key actors at each level in the system (Reagan 1972). Absolute control, as it has existed 
in the more traditional hierarchical program model, is no longer imposed from above by 
the state. 
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in. ACCOUNTABILITY IN THE IDEAL SYSTEM 
Key Actors Revisited 

While the new Texas workforce system and key elements of the accountability 
system described above are embedded in state law, certain federal laws also apply, 
particularly the Job Training Partnership Act, where performance measurement and the 
concept of evaluating return on investment are well-established. Together, these legal 
requirements set the framework for an outcomes-based accountability system for Texas. 

In an integrated workforce system like that envisioned in state law, performance 
management and evaluation functions — like the other management functions — are shared 
among a number of state agencies, as well as with their local counterparts across the 
state. In such an interdependent system of shared responsibilities, the actors have 
primary but not exclusive roles. Moreover, their responsibilities cannot be neatly sorted 
along state lines or between the state and local levels. Adding to the confusion, the role a 
particular actor is playing at any given time often depends on where one is in the 
accountability cycle; the lead and supporting roles inevitably shift back and forth as the 
process unfolds and the focus changes. 

Further, it is important to reiterate that performance is always measured and 
results evaluated in terms of expectations, either implicitly or explicitly stated. Ideally, 
these expectations are expressed as mission, vision, goals and objectives statements and 
their related success measures. All are products of strategic and operational planning. In 
short, in the ideal integrated management system, strategic and operational planning, 
performance management and evaluation are inextricably linked. 

Sorting Lead and Support Responsibilities 

At the State Level Under existing law, a number of actors have key performance 
management and evaluation responsibilities. At the state level, the Governor’s Office 
(including the governor himself), the Legislature, the LBB, the State Auditor’s Office, 
TCWEC and state agencies managing workforce funds, all have important roles to play 
assessing performance and evaluating results. This latter group includes the Texas 
Workforce Commission (TWC), Higher Education Coordinating Board (HECB), Texas 
Education Agency (TEA) and the Texas Department of Commerce (TDoC). 

The Legislature, through its Workforce Development Legislative Oversight 
Committee, as well as the State Auditor and the State Comptroller of Public Accounts 
(especially its Texas Performance Review) — though not formally part of the state’s 
workforce system — also have strong and continuing interests in performance 
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management and evaluation issues and in the results of the state’s workforce 
development efforts. 

All of these agencies are highly dependent on reliable information to carry out 
their responsibilities. Some of these data they generate on their own, while the rest must 
come from their partners in the workforce system. SOICC works with the state agencies 
generating the primary data and plays an important supporting role, gathering, formatting, 
and disseminating information throughout the system. 

Strategically, the key actors at the state level, other than the Legislature and the 
Governor, can be sorted along lines of their, responsibilities into two groups: those with 
lead responsibilities for strategic planning, budgeting and evaluation; and those agencies 
charged with managing workforce funds to achieved the desired performance and overall 
system results. 

The Governors’ Office, LBB and TCWEC. Functioning above the state agencies, 
the Governor’s Office, LBB and TCWEC share lead responsibility for developing 
strategic objectives, core measures and standards for measuring performance and 
evaluating results in the state’s workforce system. This process should be part of the 
biennial strategic planning and budgeting process and include representatives of the 
relevant state agencies and their local counterparts (e.g., local workforce boards and their 
primary service providers, public schools community colleges, education co-ops, etc.). 

The State Auditor’s Office plays an important supporting role here as well. The 
State Auditor certifies performance measures which are established for the various 
programs, including workforce development, determining they can be and are being 
measured accurately. Typical ratings issued in their regular Audit Reports on 
Performance Measures series include: certified as accurate; certified with qualifications; 
inaccurate; and unable to certify due to various problems (e.g., math errors, definitional 
problems, wrong time period, etc.). 

The resulting strategic plan, specifically the service objectives, is the yardstick by 
which performance and results are assessed and the statewide workforce system is 
ultimately held accountable. It provides the foundation for all of the other elements in the 
performance management and evaluation system, i.e., the measures, standards, rewards 
and sanctions policies and the supporting data requirements. 

These three entities also have responsibility for ensuring that each state agency 
represented on TCWEC develops operational plans (with complementary performance 
measures, standards, rewards and sanctions) and budgets for managing their workforce 
funds consistent with the strategic objectives and core measures. Further, they have 
responsibility for reviewing and approving each agency’s operational plan and budget for 
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providing workforce services, including their proposed performance measures, standards, 
rewards and sanctions policies and supporting data requirements. 

Finally, they participate in regular reviews of the system’s progress toward 
meeting performance expectations and review all evaluation findings. In the evaluation 
arena, they should focus on returns on results of public investments in workforce 
development and makes recommendations for system and services modifications to the 
Governor, the Legislature, the state administering agencies or local workforce boards, as 
appropriate. 

There are some, responsibilities at this, level which fall primarily to TCWEC. 
Specifically, they should work with the state agencies to develop joint planning 
guidelines for use by local workforce boards, as well as related planning information for 
other workforce entities involved in strategic and operational planning. TCWEC is also 
responsible for reviewing local workforce plans and makes recommendations to the 
governor for approval. Finally, under federal law, TCWEC is responsible for advising 
the Governor on performance measures related to federal workforce funds managed by 
those state agencies represented on the Council. All of this should take place in the 
jointly developed framework outlined above. At this level in the system there are both 
performance management and evaluation responsibilities. 

In this ideal system, the Legislature would appropriate funding for workforce 
services through TCWEC, consistent with the state’s strategic plan and the operational 
plans of each of the agencies managing workforce funds. TCWEC would function much 
like the current Health and Human Services Council (HHSC) with some authority to 
modify agency plans and budgets to accomplish the service objectives of the system. In 
addition, rewards (financial incentives) and sanctions would be tied to performance and 
evaluation results at this level. A small amount of the total state appropriation for 
workforce services, maybe five or ten percent would be reserved at this level to reward 
state agencies and their local counterparts. Under this approach agencies, for the first 
time, would receive at least some of their funding based on performance rather than on 
the numbers they enroll — a radical idea whose time has come. 

The State Agencies. Within the strategic framework above, state agencies 
managing workforce funds have the lead in setting operational objectives, performance 
measures and standards and develop operational strategies consistent with the state’s 
strategic objectives and core measures. As administering agencies they review local 
plans and make recommendations to TCWEC for their approval or disapproval and 
regularly monitor state and local operations to ensure that state performance expectations 
are being met. 
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The service accountability function here is clearly one of performance 
management, developing operational objectives and core services measures, monitoring 
and assessing performance of the agency and its service providers. Undoubtedly, there 
are numerous evaluative judgments related to performance, however, the primary focus at 
this level is on efficiently and effectively providing services called for by the agencies 
and their local counterparts. 

At the Local Level Under state law, local workforce boards have strategic 
responsibilities for planning, overseeing and evaluating all workforce development 
activities in their areas. As such, they are clearly the lead actors in the state’s 
performance management and evaluation system at the local level. As part of the larger 
state-local system, however, they set objectives and performance expectations for the 
local workforce system within the state’s pre-established objectives and measures 
framework which they were involved in developing. (Note: at this point the state 
framework includes the strategic core measures and operational measures set by the 
various state agencies administering workforce funds, if any. To not overly limit local 
flexibility, state agencies should not add measures unless they are specifically required by 
the funding sources.) Given the fact that we have a decentralized system, local boards are 
free to add additional objectives and measures so long as they complement state 
objectives. They should also be allowed to make adjustment to state standards for their 
areas within ranges set by the state. 

In addition to strategic responsibilities, local boards also have operational 
planning, contracting and performance management responsibilities for the workforce 
funds they directly control — e.g., JTPA, JOBS and Food Stamp E&T, but not Career and 
Technical or Adult Education. 

Under this arrangement, the performance management and evaluation 
responsibilities of the local boards are both strategic and operational. First, they should 
assess performance and evaluate results of all workforce developments activities in each 
of their areas in terms of their own strategic objectives and measures. Second, they must 
more narrowly focus on the performance of those service providers supported with 
funding directly available through the local boards, i.e., those marketing services to 
employers, operating the career development centers or providing developmental services 
such as basic education and skills training to local residents referred from the career 
centers. 
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Goals, Objectives, Measures and Standards 

There are a number of key principles that should guide the development of goals, 
objectives, measures and standards in an integrated workforce system like that described 
here. 

1) A single goals/objectives/measures hierarchy should govern the planning and delivery of 
all workforce development services in the state. An overarching set of strategic goals, 
objectives, measures and standards should be established to guide the planning activities 
of state workforce agencies, local workforce development boards and their operating 
entities. 

2) Operational goals, objectives, measures and standards should then be developed within 
the state-established framework. To ensure the “interconnectivity” of the system, 
standards should be clearly tied to specific measures which are tied to objectives which 
are, in turn, tied to goals. 

3) Fewer goals, objectives, measures and standards are far better than more. Multiple goals 
always call for multiple objectives, each leading to its own measure and standard. The 
result is too many diffemt expectations which inevitably obscure and divert attention 
from the system’s primary mission. 

4) Rewards and sanctions should be clearly connected to a common set of measures and 
standards which are used to both assess performance and evaluate results. 

5) All of the key stakeholders should be actively involved in the strategic planning process 
at both the state and local levels, but the process should be dominated by the system’s 
customers, the employers and residents of Texas. 

6) The planning terminology should change. It is no longer a discussion about funding for 
discrete programs and clients, but about financing a system that provides services to 

customers. 

Adhering to these principles should allow the Texas workforce system to develop 
and implement a much better set of goals, objectives, measures and their associated 
standards. 
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IV. A BRIEF ASSESSMENT OF THE CURRENT SYSTEM 



Based on a recent assessment conducted by Center researchers this fall (King et 
al. 1996), the current accountability system for Texas workforce development services 
can be characterized as: 

• programmatic and categorical, with the exception of recent efforts to develop and 
implement core (cross-program) performance measures; 

• federally driven for the most part in terms of federal legislative and regulatory 
requirements; 

• focused primarily on performance management rather than evaluation — and the 
two are virtually unrelated at present; 

• founded upon an artificial and inappropriate distinction between systems and 
program goals established in the existing workforce strategic plan; 

• suffering from roles/relationships confusion among the major state workforce 
partners (e g., TCWEC v. TWC, Governor's OfFice/LBB v. TWC and other 
administering agencies); 

• disproportionately focused on process not outcomes or impacts, again except for 
the recent initiative to implement core performance measures and the 
longstanding JTPA and ES efforts, as well as more recent ones for postsecondary 
Career and Technical Education; 

• overwhelmingly emphasizing participant results to the near total exclusion of 
employer-oriented measures and management/evaluation mechanisms; 

• evaluations, where they have been done, they have been ad hoc and their findings 
poorly integrated into mainline state policy decisionmaking at the highest levels, 
e g., the Texas JOBS Evaluation (King et al. 1994, Schexnayder and Olson 1995); 

• based on program/participant data collection and reporting systems which are tied 
to federal requirements and mechanisms for tracking longer-term results which 
are cobbled together and financed on an ad hoc, program-by-program basis (e.g., 
SOlCC's Automated Student Learner Follow-up System described in Froeschle 
and Anderberg 1994, 1995); 

• very uneven across the various programs and agencies: accountability systems — 
especially those for performance management — are well developed and feature 
both process and outcomes elements in both JTPA and Career and Technical 
Education (e.g., HECB 1996a-d), but are altogether lacking in the others. These 
two efforts reflect far more comprehensive thinking and approaches to 
accountability than efforts elsewhere, yet these are still program-based and not as 
well tied to true impacts as they need to be. 

With a few notable exceptions, the current workforce accountability system is 
deficient in many important respects. Despite all of the efforts made and funds expended 
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to build and maintain workforce accountability systems to date, there is vast room for 
improvement, especially if Texas is "[to] develop a state/local strategic planning, 
evaluation and accountability system ..." (TCWEC 1994). 

Key Actors, Their Roles and Responsibilities 

There are numerous conflicts and incongruencies inherent in the existing 
workforce development system’s policies and structure. These stem both from the lack 
of comprehensive workforce reform at the federal level, as well as from the framework 
established by SB 642 and HB 1863. Some of the more important of these are 
highlighted here. 

TCWEC v. Workforce Operating Agencies. In 1993, SB 642 established 
TCWEC as an autonomous state agency, complete with its own appointed board, to serve 
as one of the nation’s first overarching human resource investment councils authorized 
under the 1992 JTPA Reform Amendments (and corresponding changes to related federal 
laws). TCWEC, in this incarnation, was essentially placed above the various operating 
agencies for purposes of planning and evaluation, an appropriate structural relationship. 
However, HB 1863 subsequently “demoted” TCWEC to being just another arm of the 
Governor’s Office, even while retaining its key strategic planning, performance 
management and evaluation functions, and consolidating many of the workforce 
programs and services within the newly created TWC. The net result of these 1995 
changes has been that TCWEC in many important respects lacks the requisite power and 
standing in Texas government to carry out its functions effectively, especially those 
pertaining to evaluation. This is a general problem affecting all of its relationships with 
the partner operating agencies, but no where is this conflict more apparent than in its 
relations with the Higher Education Coordinating Board and its responsibilities for 
evaluating Career and Technical Education in the state. 

The Coordinating Board, since the late 1980s, has developed a relatively 
comprehensive, multi-dimensional system for assessing and working to improve the 
performance of Career and Technical Education efforts across the state. The federal Carl 
D. Perkins legislation also specifies a number of performance management and 
evaluation responsibilities which are to be carried out by the entity/entities administering 
such efforts within the state; in the case of postsecondary education, this entity is the 
Coordinating Board. The policy structure under HB 1863 then sets up a largely 
unworkable environment within which TCWEC is to evaluate all workforce development 
services systemically, including those already being examined via the Coordinating 
Board’s mechanisms. 
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SOICC. The Texas SOICC, as indicated above, has been very proactive in terms 
of generating labor market information and related tools and performing administrative 
data linkages and the necessary tasks needed to support more systemic evaluation of the 
state’s workforce development system. It was particularly well suited for performing 
such tasks when it was made part of the original TCWEC in 1993. Once it became a 
subpart of the newly created TWC in 1995, however, its status became problematical. 
For SOICC to serve as an autonomous entity which is regarded as fair and objective in 
performing its data-related functions in support of evaluation, it needs to be truly outside 
the orbit of any of the operating workforce agencies. Subsumed under TWC as is now 
the case — despite the fact that its day-to-day relationships with TWC is, in practice, more 
that of an independent project utilizing TWC more as its fiscal agent — this status is 
severely compromised. 

In addition, SOICC continues to secure the revenues it needs to operate the Texas 
Automated and Student Learner Follow-Up System, as well as it Consumer Report Card 
Project, on an ad hoc basis from each of the workforce operating agencies and programs. 
These efforts, which are vital to the systemic evaluation process, are also compromised 
by SOICC’s inherently dependent, service provider relationship with them. 

State/Local Relationships. One of the more important incongruencies affecting 
the state agencies’ — especially TCWEC’ s — relationships with the local workforce boards 
is that HB 1863 envisions their being held accountable for all the workforce services 
provided within their workforce area. However, given that only the “second-chance 
programs (e.g., JTPA, JOBS) have actually been placed under their direct control, they 
are being put in a very difficult position: to what extent and through what mechanism can 
they be held accountable for services which are beyond their control? Should these 
boards be brought to account for Career and Technical or for Adult Basic Education, 
efforts which they can only influence at best? These influence/control issues are at the 
heart of the performance management and evaluation system. 

Workforce Goals and Objectives 

The state’s first strategic framework for workforce development services — 
statements of mission, goals, objectives and core measures — was included in the 
Council’s strategic plan and approved by the Governor in June 1994. This common 
framework, updated after the passage of House Bill 1863 and again approved by the 
Governor in 1996, is designed to guide the Council, state workforce agencies and their 
local counterparts as they work together to develop an integrated workforce system that 
provides quality services to Texas employers and residents — services that help make 
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employers and residents more competitive in the global economy (Figure 3). As 
previously noted, together, these statements of mission, goals, objective and performance 
expectations make up the first and most important element in the state’s performance 
management and evaluation system. 



Figure 3 

State Workforce Goals 

Systems Goals 

Goal 1. To develop a statewide system supporting local career development centers where all student, 
workers, clients and employers can conveniently access a network of information and services responsive 
to their individual needs. 

Goal 2. To develop a state/local strategic planning, evaluation and accountability system for the state’s 
workforce development programs and activities. 

Program Goals 

Goal 3. All Texans will have the literacy, basic education and basic workplace skills necessary for 
educational and career advancement. 

Goal 4. Participants/workers will acquire the occupational skills to meet workplace requirements for 
long-term employment and work toward sustaining employment in high-skill, high-wage occupational 
areas. 

Goal 5. All youth will be prepared with the knowledge, skills and behaviors necessary to make the 
transition into meaningful, challenging and productive pathways in high-skill, high-wage careers, and for 
life-long learning. 



Core Workforce Performance Measures 

The Council also led the multi-agency effort which produced eight (8) core 
performance measures (Figure 4), “an initial set of measures based on the outcome- 
oriented goals and objectives set forth in the [Council’s] strategic plan” (TCWEC 1996). 
Five (5) of these measures have been approved and are in varying states of 
implementation, by legislative rider in the case of TWC and by Council requirement, as 
approved by the Governor, for the other state agencies on the Council. They have also 
been included in SOICC’s follow-up system. The remaining three have been approved 
but not yet specifically defined. Rather than align each of the measures with its 
appropriate goal, they are presented in the Strategic Plan in categories — the first five with 
definitions approved, the next three pending, and last two under development for future 
Council consideration. 





Center for the Study of Human Resources 
LBJ School/The University of Texas-Austin 



Figure 4 

Core Performance Measures 

LABOR MARKET OUTCOMES 

1. Entered Employment Measure - This measure reflects the number or percentage of people who were 
employed at the conclusion of the programs. It attempts to assess the extent to which the participant’s 
employment was a result of program services and efforts. TWC has operationalized the definition as 
follows: 

• Entered Employment Rate — percentage of program participants who, after exiting the program, secure 
employment and have wages reported in the Unemployment Insurance wage record system during or 
after the full quarter subsequent to the exit quarter. 

2. Earnings Gains Measures - These measures reflea earnings gains at designated periods. Earnings 
rates and gains reflect both the quality of jobs participants were placed into and the extent to which the 
training they received prepared them for advancement in the labor market. Two of these measures also 
have been operationally defined by TWC, as follows: 

• Earnings Gains Rate Based on Previous Earnings 

• Earnings Gains Rate Based on Program Entry — "The difference in average post-service earnings rate 
and average earnings rate at program entry"... measured by UI wage records for a period of "one 
program year beginning with July 1 ...." 

• Average Earnings Gains Based on Previous Earnings — "average change in earnings of post-program 
participants at three and five years after program exit" ... measured by UI wage records for each 
program year. 

• Average Earnings Gains Based on Program Entry 

3. Employment Retention Measure - This measure reflects whether participants in the programs have 
retained employment over time. This measure provides information about the extent to which the skills 
learned were appropriate to the labor market, i.e., whether those skills are in demand as evidenced by 
continued employment. Operationally defined by TWC as: 

• Employment Retention Rate — "percentage of individuals who entered employment subsequent to 
participating in a workforce development training program who are employed (by the same employer 
or another employer) one year after entering employment." [It is unclear whether this is measured by 
UI wage records or other data.] 

LEARNING OUTCOME 

4. Educational Achievement Measure - These measures reflea whether the experience of participants 
resulted in a degree, license, or credential. Research indicates that, on average, the higher the educational 
degree, the higher the wage. 

• Education Achievement Rate 
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Figure 4. Core Measures ... (cont.) 

OTHER SIGNIFICANT MEASURES (non-outcome) 

5. Access/Equity Measures - These measures reflect the extent to which special populations (e.g. 
minorities, individuals with disabilities) are able to gain access to and receive service from various 
programs. While the outcomes for these segments of the population are also examined as part of the 
performance measurement system, it is important to ensure they have access to the programs to begin with. 
The following measures have yet to be fully operationalized. 

• Participation Equity Rate 

• Target Population Successful Outcomes Rates * - 

• Target Population Group Identifiers for all Programs (gender, date of birth, racial ethnic groups, 
disabled individuals) 

• Target Population Group Identifiers to be reported on if currently collected. 

LEARNING OUTCOMES (definitions pending) 

6. Skill Attainment Measu res - Apart from degrees, these measures reflect whether participants attained 
the skills they were supposed to have attained in the program. Measurement of skills would require the 
development of assessment instruments, standards and curriculum for imparting the basic literacy, 
workplace or occupational skills. Operationally defined by TWC as: 

• Basic Skill Attainment: literacy and workplace basics. 

• Occupational Skill Attainment — the "percentage of certificates received by participants that are 
certified by a third party. Participants are individuals who participate in workforce development 
training programs that award certificates upon completion." 

7. Program Advancement Measu res - This measure reflects whether a participant progressed to a 
higher level of education of training following program participation. It is important to acknowledge this 
type of outcome to encourage some groups of participants, such as English as a second language and basic 
literacy students, to participate and to encourage program operators to serve less skilled participants. 

• Advancement to Next Level of Training or Education 
CUSTOMER SATISFACTION OUTCOMES (definition pending) 

8. Customer Satisfaction Measures - These measures are more qualitative in nature and allow both 
participants and employers who hire (or do not hire) participants to indicate their level of satisfaction, the 
elements of their satisfaction, and their recommendations for program improvement. Although pending, 
TWC has operationally defined these two measures as follows: 

• Employee/Client Satisfaction: — the "percentage of total client population not filing a complaint, i.e., 
clients are satisfied to the extent they did not complain." 

• Employer Satisfaction: — "Retention and Earnings Gains Rates will serve as proxy measures...." 

MEASURES FOR FUTURE CONSIDERATION 

9. Training-Relatedness of Placement - This measure is being tracked during the initial pilot phase. 

10. Program Cost Effectiveness - This measure is being researched prior to developing a definition. 
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These core measures arose both from the Council's interest in developing and 
instituting performance measures which would cut across the various workforce 
programs then in existence, as well as from the related interagency effort led by the U.S. 
Department of Labor (following passage of the 1992 JTPA Amendments) to coordinate 
data collection among the federally funded programs and the National Governors' 
Association (NGA) project to develop core and common data elements for performance 
measurement for these same programs (Trott and Baj 1996). Not only was Texas one of 
the handful of states participating in the NGA project, but it stepped quickly into the lead 
role on developing core performance measures. - This has important implications for the 
task at hand: the current set of core measures (1) were largely developed and approved 
prior to the passage of HB 1863, and (2) were intended to be based primarily on existing 
data collection and reporting systems common to Texas workforce agencies and 
programs. 

As things now stand, the core performance measures are enjoying a very uneven 
reception, giving the workforce system confusing signals at both the state and local level. 
Some notable examples of this unevenness and confusion include the following: 

• The 74th Texas Legislature explicitly incorporated the eight approved core 
measures into TWC's portion of the General Appropriations Act in 1995 (House 
Bill 1, Article IX, Section 166), but not those of the other major workforce 
agencies affected by HB 1863; 

• TWC's Workforce Development Board Planning Guidelines (TWC 1996) 
augmented TCWEC's five strategic goals with a "Work-First" Goal of its own — 
"Increase the percentage of Texans who become and remain independent of 
public financial assistance," which it led with — and folded the five operationally 
defined core measures into the guidelines as well; two of these were explicitly 
identified as "key" measures: the Entered Employment Rate, and Earnings Gains; 
and 

• To date, LBB has not included any of the core measures into its own set of 33 
"Key Measures" for TWC or any of the other agencies, including the Council. 
Most of the measures which LBB has proposed for adoption and use are either 
process-oriented measures or are highly program-specific (e.g., Communities-in- 
Schools). 
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V. MAJOR GAPS, CHALLENGES AND NEXT STEPS 

There are a number of major gaps in the current accountability system. Some of 
the more important of these gaps are presented here, followed by a brief discussion of 
challenges and next steps required to move from the current to the more ideal, systems- 
oriented approach to evaluating workforce efforts in Texas. 

Major Gaps and Challenges 

Major gaps characterize the current accountability system, relative to the ideal one 
laid out above. The marble cake analogy is useful to keep in mind when considering the 
following gaps and the challenges they present. 

Workforce Service/Agency Boundary Issues. Several major efforts for providing 
workforce services to residents and employers still operate largely outside the immediate 
boundaries of the workforce system created under SB 642 and HB 1863, including 
especially Career and Technical Education (HECB, TEA), Adult Education (TEA) and 
the Smart Jobs Fund (TDoC). Not that these efforts are not essential elements of the 
system, but that they were not among the 28 programs consolidated into TWC with 
funding slated to pass down to local workforce boards. They function beyond the direct 
control of TWC but are under the strategic planning, oversight and evaluation umbrella of 
TCWEC. This is a state as well as a local issue. While the Texas workforce system 
clearly can continue under this construct, important adjustments must be made to the 
expectations and the practice of accountability at the state and local level if it is to do so 
effectively. 

At the state level, the problem is evident in the fact that the Legislature in 1995 
directed that the core performance measures applied only to TWC (House Bill 1) and not 
the other large programs operating under HECB, TEA and Commerce. For the upcoming 
Biennium, even the 33 "key measures" accorded to TWC exclude the core measures. In 
addition, the strategic goals, objectives and core measures developed by TCWEC for the 
entire workforce system under the mandates of SB 642 and HB 1863 were only placed on 
TWC efforts by the LBB/Govemors' Office under its biennial process; again, the other 
large efforts referred to above — though they have made their own efforts to coordinate in 
the process — have not been accorded the same expectations and measures. This creates a 
serious imbalance at the state level. 

This imbalance has ramifications for local workforce boards and service delivery 
as well. Under the existing parameters, it is not at all clear that local workforce boards 
can be held accountable in exactly the same way for the performance of workforce 
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services whose funds fall largely outside their control. A local board might be able to 
leverage career and technical services from local community colleges, and they may even 
be able to work out ingenious service collaboration agreements. However, these boards 
cannot directly control these services; they can only attempt to influence their delivery. 
Thus, while all of the workforce system goals and objectives and measures and standards 
should ultimately be extended to all of these services, regardless of which state/local 
entity provides them, it may be both prudent and pragmatic to consider instituting a 
process which rewards state operating agencies and local boards richly for outstanding 
service, but does notsanction them for failures which they cannot avert. 

A very different alternative would be to create a combined state-level, workforce 
incentive fund (say, approximately 5-10 percent of the total) at TCWEC to use for 
rewarding state agencies and their local counterparts — e.g., local workforce boards, 
community colleges, adult education cooperatives, etc. — for their contribution towards 
achieving the key state workforce goals and objectives. 

SOICC's ongoing Consumer Report Card project, which is being conducting in 
Texas with U.S. Department of Labor funding, offers yet another alternative. The 
Consumer Report Card effort operates on the presumption that one of the primary means 
of influencing behavior in a market-based system such as ours, is to provide fairly 
detailed, easily accessible information on the performance of specific providers in the 
system. Consumers can then make their own judgments — aided in part by staff in the 
career service centers if desired — based upon more complete provider performance 
information. The control/influence problem arising from the service/agency boundary 
problem may be minimized via this approach, particularly for the more knowledgeable 
consumers. Regardless of the approach, if sanctions are ultimately called for, they should 
apply to boards, providers and related entities; workforce funding and services still must 
remain in the area for serving its residents and employers. It may mean that a new set of 
boards/providers is required in order to do a better job. 

Last, technical assistance and training (TAT) both serves as the necessary front 
end to a sanctions process and as an important part of the workforce capacity building 
process. Before invoking sanctions on poor performers, it is appropriate to offer local 
workforce boards TAT so that they have an opportunity to make improvements on their 
own. TAT also can contribute to improved performance over time by assisting locals in 
organizing more appropriately, discovering and implementing more effective workforce 
service interventions, etc. There are many precedents for such capacity building 
initiatives, including the Manpower Institutional Grantee system in the 1970s and early 
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1980s, as well as the Education Service Centers and even the Agricultural Extension 
Centers beyond the workforce area. 

Strategic Plan Revisions. As suggested above, a number of areas in the existing 
Strategic Plan developed by TCWEC for the workforce system need to be improved. The 
existing split between "system" and "program" goals and objectives does not appear to 
conform to the system/services notions outlined here. Instead, they seem to relate more 
to a "management system" (e.g., fiscal, MIS) versus "categorical program" distinction 
which may not be as useful. In addition, many of the existing goals appear to be written 
more in the form of vision- statements,, while some of the objectives read more like 
strategies, and the core measures do not align well with the current goals and objectives. 
Finally, Texas' workforce legislation requires local service delivery systems to have both 
resident and employer services components. Implicit in this requirement is the 
importance of employers as a primary customer of the workforce system. The current 
plan fails to include a single employer-related goal or objective. This must clearly be 
addressed in the very near future, in that measures and standards for both performance 
management and evaluation should be inextricably tied to goals at the state and local 
level. 

TCWEC. TCWEC is functioning at a disadvantage at present in light of the 
performance management and evaluation responsibilities accorded it in existing law. 
TCWEC would be more appropriately situated to carry them out effectively if it were 
either a completely independent and separate entity (e.g., the Health and Human Services 
Commission) or part of the Governors' Office of Budget and Planning. It is inappropriate 
for the agency which is legislatively entrusted with the ultimate responsibility for 
planning, overseeing and evaluating the performance of the entire workforce system to be 
relegated to a relatively subordinate position in the Governor's Office, while the agencies 
it oversees answer to powerful, independent boards and commissions. Such a structural 
mismatch in governance creates unrealistic expectations in terms of an accountability 
system and what it can achieve. 

SOICC. Over the years, SOICC has emerged as a key actor in documenting 
service outcomes — especially those over longer periods which require substantial 
administrative data matching — for both residents and employers across the existing array 
of workforce programs (Froeschle and Anderberg 1994, 1995). However, SOICC lacks 
autonomy from the operating agencies, not to mention reliable, ongoing sources of funds, 
both of which are required if it is to continuing serving as the autonomous, objective 
descriptor of the workforce system's performance for Texas. Without SOICC's 
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leadership, vision and persistence, these data would in all likelihood not have been 
available to state policymakers and evaluation researchers. 

Under SB 642, SOICC and its duties were transferred to the newly created (then- 
independent) TCWEC though in practice it functioned largely as its own entity. Under 
HB 1863, it is part of TWC, despite the fact that, by federal law, its governing board is 
comprised of officials from nonTWC programs (e.g., career and technical education). In 
addition, virtually none of SOICC's budget for performing the essential data collection 
and reporting functions in support of both performance management and evaluation is in 
the form of "hard money" contributed by the operating agencies. Instead, SOICC has 
been aggressively entrepreneurial in raising these funds — a large share of them are 
federal— each year on a contract-by-contract basis. This is an inappropriate way to 
conduct or provide for one of the most critical parts of the performance 
management/evaluation system. The result of this situation is that SOICC s data 
collection and reporting efforts are of necessity designed more for its individual 
customers (HECB, TWC, USDOL) and their needs than for the workforce system's 
evaluation requirements. 

SOICC’s autonomy should be restored, and its funding should be made more 
dependable. It should not be put in a position of having to sell the operating agencies its 
information services, at least as they are required to support the evaluation of workforce 
services. 

Core Performance Measures Revisions. The core performance measures which 
have been developed and adopted (1995) for use by the Texas workforce system need to 
be modified in several key respects as well. First, it is clear from this analysis that, while 
employers are quite clearly one of the system's primary customers, employer-related 
measures are lacking (as are associated employer goals and objectives). Such measures 
have received very little space in the evaluation or performance management literature 
and do not offer much to go on in terms of a track record. Employer goals, objectives 
and related measures must be developed in the near future; suggested employer measures 
will be offered in the future. The role of skills standards must also be explored further for 
possible connections to core measures, though this tie may be premature just yet, skill 
standards development appears to be progressing on a slower track at the national and 
state level and might be considered for future pilot projects related to employer-oriented 
performance measures. 

Second, while customer satisfaction has been much touted as a measure of 
performance in the management literature since the early 1980s, it is unclear that it 
should play a role at the state level in the new system. What are the implications of a 
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decline in employer or resident satisfaction at the systems level? What responses are 
called forth? Such measures have an essential place in measures at the local level, closer 
to the point at which services are actually being delivered. To be useful, customer 
(dis)satisfaction measures should elicit relatively immediate responses in terms of service 
delivery changes locally. Customer satisfaction should be dropped from the core 
performance measures at the system level, however, such measures should be featured as 
an integral part of the local performance measurement array. 

Third, on further examination, program advancement should be eliminated from 
the existing set of core; measures and relevant -portions contained within it combined into 
the educational skills attainment measure. Program advancement, per se, does not appear 
to warrant core-measure status. It is not a measure well recognized by employers. After 
all, it is skills-related to employers' needs that are important to them. Instead, the 
program advancement measure appears to persist in the longstanding (but ill-advised) 
tradition of process-oriented measures which SB 642 and HB 1863 attempted to rectify. 
If the system is to be truly performance — and primarily outcomes — driven, then program 
advancement measures should be dropped in favor of ones tied to skills attainment. 

Fourth, earnings gain measures based on earnings at program entry should also 
be eliminated. Considerable discussion and debate led to their inclusion in the original 
set of core measures. However, there is a "fatal flaw" inherent in these measures on the 
one hand, and, on the other hand, more recent developments have made them obsolete. 
The fatal flaws stem from the fact that most participants and many students seek to enroll 
in workforce related services due to a recent or imminent decline in their earnings. These 
declines— referred to in the literature as the "earnings dip" phenomenon (Ashenfelter 
1978; Bassi 1984) — are usually temporary, and much of their subsequent earnings gain 
represents a natural recovery to their longer-term earnings path. When changes in 
earnings are measured from program entry, they tend to attribute far too much of the gain 
to the intervention. Such measures also reinforce service providers' tendencies to 
"cream," enrolling those most likely to do well regardless of the intervention. Happily, 
the recent availability of up to four years of UI earnings records on-line means that, as 
long as Social Security numbers (SSNs) are used for participant/student identification, it 
will be possible to secure sufficient earnings data to avoid the earnings "dip" problem. 
Note that this solution works primarily for adults and youth with substantial earnings 
history; it does not work for new entrants whatever their age. Of course, neither does 
measuring gains based earnings at program entry. This measure should be dropped. 

Fifth, the initial development of the access/equity measures appears to have 
generated measures for far too many groups to be practical, particularly in the context of 
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a system intended to offer access to all. At the broadest level, universal access implies 
few if any service priorities. In reality, however, access/equity concerns relate more to 
needs that certain groups may have (e.g., basic skills deficiencies, inadequate earnings, 
poor work experience). The current measures exhibit a "second-chance program" 
emphasis which should be reconsidered. 

Moreover, if all of the groups for whom core access/equity measures have been 
identified in the current set are truly priority groups of interest, then ultimately none of 
them are. This is a lesson learned the hard way over many years of implementing 
workforce programs. It is important that, policymakers prioritize a smaller number of 
groups for inclusion in the access/equity measures — and to consider them in the context 
of particular service strategies — if they are to serve their purpose effectively. This 
process has already begun with the 1996 LBB/ Governor's Office process of establishing 
goals and measures for agencies. A final point on the access/equity issue is that there 
may need to be a companion set of employer-oriented measures. For example, to some 
extent, a small-employer penetration rate for business services might be thought of as an 
access measure on the employer side. 

Finally, a technical correction should also be made to the earnings gain measures. 
The current earnings measures improperly measure gains by computing them based on 
differing cohorts of participants. For a gain measure to accurately capture earnings gains, 
the cohort being tracked should contain the same members both pre- and post-services. 
Note that TWC soon will have up to 16 quarters of UI wage records available on-line at 
any time, facilitating measurement of earnings (and employment) measures considerably. 
Accurate earnings (and employment status) measurement requires much longer 
measurement periods than have typically been feasible (King et al. 1996). 

TCWEC and its partners in the workforce system must push ahead with further 
research and design work on several measures. SOICC's preliminary analysis details the 
complexity of devising and using training-relatedness measures as indicators of success. 
Cost-effectiveness measures also need further work, including the development and 
testing of some type of retum-on-investment (ROI) measure. This would have been a 
more problematic measure under the program-oriented world which existed prior to HB 
1863's passage, but, data systems permitting, it would appear to be one of the essential 
measures (King 1995). From the systems level, ROI or other cost-effectiveness measures 
might be most appropriately tied to the more intensive, development-oriented service 
strategies. Employers are likely to relate better to a workforce system which can 
communicate in an ROI context as well. Another set of measures which need intensive 
research are those tied to skill attainment. Current measures appear to be poor proxies for 




26 36 



Center for the Study of Human Resources 
LBJ SchooLThe University of Texas-Austin 

real skill attainment. This should be tied to ongoing skill standard development work by 
TCWEC and other national and state groups. 

Recommended Next Steps 

A number of important next steps are recommended to move towards a systems 
evaluation approach to accountability for Texas workforce development services along 
the lines described earlier. These recommended steps are organized into three broad 
phases. Phase I consists of steps to be taken immediately, i.e., before June 30, 1997. 
Phase II offers a series of longer-term steps to be taken from July 1997 through June 
1999. And, Phase III presents steps which should be taken over the long run, between 
July 1999 and June 2001. 

This approach recognizes the immediate pressures to set in place some measures 
that will allow the system to respond to the elected and appointed officials who inevitably 
ask the straightforward questions, "Is it working?" and "Are we making a difference?" 
These are inherently fair and appropriate questions for them to ask, and the workforce 
system must be prepared to answer them with some degree of confidence. So, it is 
important for the system to step out and soon. 
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Phase I: Immediate (present - June 1997) 

1.1 The LBB, in partnership with TCWEC and the Governor's Office, should immediately 
establish two of the existing core performance measures — the Entered Employment Rate 
and Average Earnings Gain Based on Previous Earnings— as the key workforce 
measures for the next Biennium, not just for TWC but for all of the workforce system's 
operating agencies. These two measures should be inserted into the appropriations bill 
for all the agencies. 

1.2 TCWEC should work closely with its workforce partner agencies to design and establish 
performance "targets" for the Entered Employment and Average Earnings Gain 
measures for all the relevant workforce agencies, recognizing that such targets will 
ultimately vary by service population and regional conditions, among other factors. 

1.3 TCWEC should make the following modifications to the existing set of core performance 
measures: 

• Eliminate the earnings gain measures based on earnings at program entry. 

• Base the remaining earnings gain measures on same-cohort, pre-post 
computations, with previous earnings calculated on at least 6 to 8 pre-entry 
Ul-covered employment quarters. 

• Eliminate customer satisfaction measures at the state/systems level, while at 
the same time encouraging local boards and other entities to continue using 
them in their performance management systems locally. 

1.4 TCWEC should continue working with its partner agencies to conduct further research 
designing the remaining core measures, attempting to resolve problems identified with 
access/equity, refining the relationship between program advancement and related 
measures, and researching and developing employer-oriented, skill attainment and cost- 
effectiveness (e.g., retum-on-investment) measures. 

1.5 TCWEC should work closely with each of its partner agencies at the state and local level 
to identify the basic data reporting elements required to support the core performance 
measures established for the workforce system. 

1.6 The Legislature should reinstitute SOlCC's status as an independent entity and provide 
for its funding, so that it may more appropriately and objectively fulfill its data support 
and coordination role for the entire workforce system. 
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Phase II: Longer-Run (July 1997 - June 1999) 

2.1 TCWEC, working with its key partners, the Legislature, the Governor’s Office and the 
administering agencies, should clarify the key workforce actors' strategic roles and 
relationships in the workforce system. 

2.2 TCWEC should redo its strategic planning process, making it inclusive of its workforce 
partners at the state and local level and substantially revising the existing goals and 
objectives as the framework for refining the core measures. 

2.3 The LBB, the Governor's Office and TCWEC should take the lead, working closely with 
the workforce partner agencies at the state and local level, to bring the state and local 
workforce planning, budgeting and funding process in line with the state's biennial 
planning and budgeting cycle. 

2.4 TCWEC, working closely with its workforce partners at the state and local level, should 
refine the remaining core performance measures, following the principle that less is 
better, in order to provide clear focus to the system. For example, TCWEC should. 

• Establish an employer-based core measure. 

• Clarify and establish an appropriate but very limited set of access/equity core 
measures, presumably tied to needs-based service strategies. 

• Clarify and establish an appropriate skill attainment core measure. 

• Clarify the relationship between the program advancement and closely related 
core measures (e.g., skill attainment). 

2.5 TCWEC should work closely with its partner agencies at the state and local level to 
identify the basic data reporting elements required to support any additional core 
performance measures established for the workforce system. 

2.6 TCWEC, working closely with each of its partner agencies, should develop and utilize 
better mechanisms to more effectively communicate its service goals, objectives and 
measures to local boards and other entities. 

Phase HI: Long-Run (July 1999 - June 2001) 

3.1 The Legislature should appropriate all federal and state workforce funds through 
TCWEC to the workforce partner agencies and provide TCWEC with limited authority to 
modify the agencies' allocations to achieve system objectives, federal law and regulations 
permitting. 
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3.2 The Legislature should reserve from 5-10 percent of appropriated federal and state 
workforce funds for rewarding state agencies and their local counterparts based on 
their contribution to achieving system performance objectives, federal law and 
regulations permitting. 
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VI. ACTION PLAN 

This section presents an evaluation action plan — with major tasks, actors with 
lead responsibility, and timelines — designed to move Texas from the current approach to 
implementation of the desired workforce accountability system. It is structured to 
accomplish this in three broad phases: Phase I, which are immediate tasks to be 
accomplished before June 30, 1997; Phase II, which are longer-term tasks to be carried 
out from July 1997 through June 1999; and Phase III, encompassing changes to the 
system which should be accomplished, over, the long run, between July 1999 and June 
2001. 



Task (Lead Responsibility) Target Date 
Phase I: Immediate (present - June 1997) 



1.1 

1.2 

1.3 

1.4 

1.5 

1.6 



Establish the Entered Employment Rate and Average Earnings Gain Based on Previous 
Earnings as the key workforce measures for the next Biennium, for all workforce 
operating agencies. (LBB, GO & TCWEC) 3/97 



Design and establish performance "targets" for Entered Employment Rate and Average 
Earnings Gain measures. (TCWEC, partner agencies) 3/97 

Modify the existing core performance measures, eliminating several and making 
technical corrections to others. (TCWEC, partner agencies) 6/97 



Design the remaining core measures, including access/equity, employer-oriented, skill 
attainment and cost-effectiveness (e.g., ROI) measures. (TCWEC, partner agencies)6/97 



Identify the basic data reporting elements required to support the core workforce 
performance measures. (TCWEC, partner agencies) 6/97 

Reinstitute SOICC's status as an independent entity and provide for its funding. 
(Legislature) 6/97 



Phase II: Longer-Run (July 1997 - June 1999) 



2.1 Clarify key actors' strategic roles and relationships in the workforce system. (TCWEC) 

10/97 

2.2 Redo the workforce system's strategic planning process, substantially revising existing 

goals/objectives. (TCWEC, state/local partners) 12/97 
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2.3 Bring the state and local workforce planning, budgeting and funding process in line with 

the state's biennial planning and budgeting cycle. (LBB, GO, TCWEC, state/local 
partners) 6/99 

2.4 Refine the remaining core measures, including employer-based, access/ equity, skill 

attainment and program advancement. (TCWEC, state/local partners) 1/98 

2.5 Identify the basic data reporting elements required to support additional core performance 

measures. (TCWEC, state/local partners) 6/98 

2.6 Develop better' mechanisms for communicating service goals, objectives and measures to 

local boards et al. (TCWEC, partner agencies) 6/99 

Phase HI: Long-Run (July 1999 - June 2001) 

3.1 Appropriate all federal and state workforce funds through TCWEC to workforce agencies 

and provide TCWEC with limited authority to modify agencies' allocations to achieve 
system objectives. (Legislature) MM/99 

3.2 Reserve 5-10 percent of appropriated federal and state workforce funds for rewarding 

state agencies and their local counterparts based on their contributions to system 
performance. (Legislature) MM/99 
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